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JOB  SKILLS  EDUCATION  PROGRAM 


TRADOC  TEST  PLAN  REPORT 

Executive  Summary 

The  Job  Skills  Education  Program  (JSEP)  Test  Plan  is  a  result 
of  a  TRADOC  initiative  to  determine  if  job  related  academic  skills 
taught  using  the  JSEP  computer  based  instruction  (CBI)  curriculum 
helps  soldiers  acquire  and  retain  job  related  skills.  The 
evaluation  of  JSEP  conducted  by  American  Institutes  for  Research 
from  January  through  April  1983  was  not  able  to  provide  sufficient 
answers  to  questions  for  TRADOC  to  adopt  JSEP  as  a  curriculum 
delivery  system  for  basic  skills  instruction.  Prior  to 
implementing  JSEP,  TRADOC  conducted  an  evaluative  study  of  JSEP 
based  on  the  following  hypotheses: 

a.  Soldiers  participating  in  JSEP  will  demonstrate 
acquisition  of  job  skills. 

b.  Soldiers  participating  in  JSEP  will  demonstrate  retention 
of  job  skills. 

We  tested  the  hypotheses  on  soldiers  in  AIT  and  120  days  after 
completion  of  training.  Our  research  into  previous  studies  on 
skill  retention  assured  us  that  120  day  retention  study  would  be 
significant. 

We  decided  on  seven  MOSs  representing  combat  arms,  combat 
services,  and  combat  services  support  at  six  TRADOC  installations. 
We  matched  soldiers  and  assigned  them  to  control  or  experimental 
groups,  with  the  experimental  group  receiving  JSEP  lessons  and 
both  groups  tested  with  an  Army  Research  Institute  developed 
Job-Relevant  Knowledge  Test  for  specific  MOSs. 

TRADOC  Analysis  Command  (TRAC)  at  White  Sands  Missile  Range 
agreed  to  analyze  the  data  we  collected  during  the  Test  Plan. 

Their  initial  analysis  of  data  in  September  1989  demonstrated 
marginal  significance  based  on  grouping  soldiers  by  MOS.  Our 
subsequent  analysis  of  the  data,  based  on  grouping  soldiers  by 
percent  of  JSEP  lessons  complete  within  the  experimental  group  and 
compared  to  our  control  group,  showed  significant  effects  between 
JSEP  instruction,  degree  of  prescription  completed  and  severity  of 
prescription  and  immediate  results  on  the  ARI  test. 

Most  of  the  data  in  the  subsequent  analysis  was  available 
during  the  initial  analysis.  The  primary  differences  in  the  . 
initial  and  subsequent  analysis  was  due  to  more  logical  grouping 
of  the  data.  Additionally,  thirteen  follow-up  test  results  were 
available  for  the  subsequent  analysis  that  were  not  received  at 
the  time  of  the  initial  TRAC  analysis. 


Inferential  statistical  analysis  of  the  data  based  upon  TRAC's 
grouping  of  results  by  MOS  was  hampered  by  too  few  degrees  of 
freedom.  Conversion  of  scores  to  percentage  allowed  us  to  view 
levels  of  intervention  and  levels  of  prescription  for  effects  and 
relationships  ignored  in  the  preliminary  TRAC  analysis. 

By  using  the  comparison  group  data,  we  were  able  to  regress  to 
a  common  start  point  for  both  the  experimental  and  comparison 
groups  creating  a  theoretical  expected  learning  regression.  Since 
ideal  conditions  are  that  every  soldier  complete  all  JSEP  lessons 
prescribed,  we  were  able  to  block  on  both  of  these  variables  in  an 
analysis  of  variance  to  extract  information  on  effects  and 
relationships  key  to  understanding  the  dynamics  of  the  JSEP 
process . 
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INTRODUCTION 


The  evaluation  of  JSEP  conducted  by  American  Institutes  for 
Research,  under  contract  from  Army  Research  Institute  from  January 
to  April  1988,  could  not  determine  what  effect  JSEP  had  on 
soldiers  job  performance.  The  purpose  for  developing  JSEP  was  to 
improve  soldiers  job  performance  yet,  at  the  conclusion  of  lesson 
development  and  the  formative  evaluation,  we  did  not  know  if  JSEP 
had  the  desired  effect  on  soldiers.  In  order  to  get  an  answer  to 
this  question  the  Education  Directorate,  TRADOC,  developed  a  test 
plan  with  the  hypotheses  that  JSEP  is  a  cost  effective  program  to 
improve  the  acquisition  and  retention  of  soldiers  job  skills. 

We  used  soldiers  in  Advanced  Individual  Training  (AIT)  for  our 
Test  Plan  since  this  was  where  all  soldier's  would  be  acquiring  a 
majority  of  the  job  tasks  required  in  their  particular  MOS.  The 
retention  portion  of  our  Test  Plan  measured  soldiers'  skills 
approximately  120  days  after  completion  of  training  when  the 
soldier  was  at  the  first  permanent  duty  station.  The  "cost 
effective"  reference  in  our  hypothesis  is  to  the  time  and  money 
required  to  provide  JSEP  to  soldiers.  In  order  for  JSEP  to  be 
cost  effective,  we  expect  to  see  measurable  increases  in  job 
performance.  Time  constraints  did  not  permit  us  to  conduct  a 
longitudinal  study  and  track  soldiers  two  or  three  years  after 
receiving  JSEP.  However,  our  research  into  previous  studies  on 
skill  retention  assures  us  that  120  day  study  of  retention  is 
significant. 

TEST  DESIGN  AND  DATA  COLLECTION 


We  decided  on  seven  MOS  to  use  in  the  test  plan:  11B ,  123, 
71L,  76Y,  88M ,  94B  and  9  5B .  The  seven  MOS  represent  combat  arms, 
combat  services,  and  combat  services  support.  Our  reasons  for 
using  these  seven  MOS  were:  over  seventy  percent  of  MOS  tasko  are 
taught  in  AIT;  schools  have  sufficient  students  per  cycle;  all 
schools  are  in  TRADOC;  ARI  has  developed  task  proficiency  tests 
(described  below)  for  MOS  that  can  be  used  as  end-of-cour se  tes  s; 
start  dates  for  AIT/OSUT  were  compatible  with  time  frame  of'  the 
test  plan;  and  PLATO  terminals  to  deliver  JSEP  lessons  were 
available  in  sufficient  quantities  for  the  number  of  soldiers 
needed  for  the  test  plan.  The  MOS  in  the  test  plan  were  at  t  e 
following  six  TRADOC  installations:  Forts  Benning,  Dix,  Jacxson, 
Lee,  McClellan,  and  Leonard  Wood. 


We  could  not  test  JSEP  on  highly  technical  MOS  because  the 
length  of  these  AIT  would  extend  the  test  plan  beyond  the 
scheduled  dates.  The  Test  Plan,  as  designed,  could  not  consider 
AIT  over  9  weeks  in  duration. 

We  used  only  CAT  Ilia,  mb,  and  IV  active  Army  soldiers.  We 
selected  two  matched  groups  of  100  soldiers  each  to  be  in  a 
control  and  experimental  group.  We  matched  soldiers  within  the 
two  groups  based  on  the  following:  sex,  race,  AFTQ  (or  GT) ,  ASVAB 
subtest  score  required  of  MOS,  JSEP  test  scores,  and  educational 
level . 

The  first  step  in  beginning  the  test  plan  was  for  installation 
education  services  officers  (ESO)  to  test  all  soldiers  in  mental 
category  III  and  IV  who  were  in  a  designated  AIT/OSUT  cycle  during 
the  period  March  -  April  1989.  All  sites  used  the  JSEP  diagnostic 
tests,  along  with  other  data  described  above,  for  matching 
soldiers  into  control  and  experimental  groups.  The  JSEP  Test  we 
used  was  a  195  item  paper  based  test  that  does  not  test  all 
prerequisite  competencies  in  a  soldiers'  MOS  prescription. 

However,  the  JSEP  lesson  developer  validated  the  test  items  for 
most  JSEP  lessons.  Since  the  JSEP  lesson  developer  never 
delivered  a  diagnostic  test  with  the  lessons,  we  had  to  use  this 
test  knowing  its  limitations. 

The  group  designated  experimental  was  given  the  JSEP 
prescription  for  their  MOS  and  received  JSEP  instruction  on  only 
those  tasks  they  are  deficient  in  based  on  the  diagnostic  test. 

ESO  working  closely  with  training  battalions  scheduled  JSEP 
lessons  for  approximately  two  hours  per  evening,  five  evenings 
each  week  of  AIT/OSUT.  Some  Education  Centers  opened  at  special 
times  to  accommodate  the  soldiers  while  in  training.  In  order  to 
give  soldiers  a  chance  to  complete  their  prescription  of  lessons, 
we  suggested  a  minimum  of  50  hours  from  the  AIT/OSUT  schedule. 

Time  made  available  ranged  from  36  to  54  hours  with  47  hours  being 
the  average. 

At  the  end  of  AIT/OSUT  the  ESOs  gave  both  the  exper  imental  '^and 
control  groups  MOS  proficiency  tests  developed  by  ARI .  ARI  refers 
to  the  19  tests  as  Job  Relevant  Knowledge  Tests.  We  used  the 
seven  tests  that  corresponded  to  the  MOS  in  our  Test  Plan.  The 
tests  measure  cognitive  components  of  formal  training  experiences, 
specifically  AIT  and  are  meant  to  predict  first  and  second  term 
job  performance.  The  tests,  even  though  just  released  by  ARI  to 
TRADOC  for  use  in  March  1989,  were  deemed  to  be  the  best  measure 
of  job  skills  taught  in  the  training  base.  According. to  ARI 
research  findings,  all  of  the  tests  have  relatively  high 
reliability  coefficients  (mean  alpha  was  .88  across  all  tests). 

Data  collected  in  1983  during  the  Task  Analysis  phase  of  JSEP 
development,  and  stored  at  TRADOC,  enabled  us  to  locate  the 
soldier's  tasks  on  which  each  JSEP  lessons  is  based.  We  then 
identified  the  soldier’s  tasks  ARI  used  to  develop  their 
Job-Relevant  Knowledge  Test  questions.  Finally,  we  matched  the 
tasks  used  for  JSEP  lessons  with  the  tasks  taught  in  AIT/OSUT. 


This  enabled  us  to  verify  the  relation  between  JSEP  lessons  and 
what  was  tested  on  the  ARI  test  with  what  was  being  taught  in 
AIT/OSUT. 

The  results  from  the  ARI  Job-Relevant  Knowledge  Tests  helped 
us  test  our  first  hypothesis  which  is:  JSEP  is  a  cost  effective 
program  to  improve  soldiers  acquisition  of  job  skills.  The 
measure  of  cost  effectiveness  was  the  time  taken  out  of  the 
training  schedule  for  JSEP  was  worthwhile  if  the  soldier  improves 
knowledge  of  job  tasks. 

One  hundred- twenty  days  after  soldiers  left  AIT/OSUT,  we 
queried  the  DA  Enlisted  Master  File  to  locate  soldiers  at  their 
duty  station  for  follow-up  testing  which  would  be  approximately 
123  days  after  AIT/OSUT.  We  sent  the  ARI  Job  Relevant  Knowledge 
Test  to  ESOs  at  the  soldiers'  duty  station  with  instructions  to 
administer  the  test  and  return  to  us  for  scoring.  We  matched  this 
score  to  the  results  of  tests  given  at  the  end  of  AIT/OSUT,  for 
both  groups,  to  see  if  those  taking  JSEP  retained  skills  longer 
than  the  control  group  that  received  no  JSEP  instruction.  This 
tested  the  second  part  of  our  hypothesis:  retention. 


RESULTS  OF  THE  TRADOC  TEST  PLAN 


At  TABLE  1  is  a  profile  of  the  experimental  and  control  groups 
that  we  collected  data  on  for  the  Test  Plan.  JSEP  lessons  were 
not  assigned  until  soldiers  entered  AIT  portion  of  training.  The 
time  allotted  for  JSEP  lessons  on  the  PLATO  computer  system  varied 
at  each  installation.  Some  installations  arranged  for  lessons  to 
be  given  during  the  duty  day,  others  in  the  evenings.  All  groups, 
except  the  71L  at  Fort  Jackson,  had  specific  hours  they  could 
work  on  lessons.  71L  had  50  hours  allotted,  but  were  given  an 
unlimited  amount  of  time  to  complete  all  of  lessons  prescribed. 
Also,  71L  received  the  full  prescription  of  MOS  lessons  which 
resulted  in  up  to  80  lessons  being  assigned. 


TABLE  1 

PROFILE  OF  EXPERIMENTAL  AND  CONTROL  GROUPS 


MOS 

EXPERIMENTAL 

N 

CONTROL 

N 

LENGTH 

AIT/OSUT 

TIME 

ALLOTTED 

MEAN  JSEP 
LESSONS 

118 

22 

22 

13  wk 

50  hr 

34 

12B 

5 

4 

13  wk 

36  hr 

26 

7 1L 

14 

17 

9  wk 

50* 

47 

76Y 

17 

10 

7  wk 

54  hr 

17 

8  8M 

24 

25 

8  wk 

44  hr 

32 

94B 

17 

16 

9  wk 

46  hr 

32 

9  5B 

4 

4 

1 6  wk 

53  hr 

22 

Total 


98 


103 


ESOs  assigned  soldiers  to  the  experimental  and  control  groups 
so  that  an  equal  number  were  in  each  group.  As  seen  in  TABLES  1 
and  2,  there  are  different  numbers  in  the  two  groups.  This  is  a 
result  of  minor  and  insignificant  mortality  between  the  time 
soldiers  were  assigned  to  a  group  and  the  first  ARI  test  at  the 
end  of  AIT/OSUT. 

TRADOC  Analysis  Command  (TRAC)  conducted  the  analysis  by  MOS 
groups  (TABLE  2)  based  on  the  results  of  the  first  ARI  Job 
Relevant  Knowledge  Tests.  TRAC  analyzed  the  experimental  group 
data  to  determine  if  there  is  a  relationship  between  the  number  of 
JSEP  lessons  completed  and  the  ARI  total  test  score  results.  It 
showed  JSEP  benefited  the  71L  MOS  personnel.  This  correlation  was 
significantly  different  from  zero  (PC. 05).  It  shows  that  soldiers 
who  completed  more  JSEP  lessons  tended  to  have  higher  ARI  total 
scores.  The  failure  to  identify  significance  by  MOS  is  in  direct 
relationship  to  the  small  n-size  artificially  created  by  analysis 
by  MOS. 


TABLE  2 

TRAC  ANALYSIS  OF  ARI  TEST  RESULTS  BY  MOS 


PERCENT 

CORRECT 

STANDARD 

DS 

CONTROL 

EX?  SR 

t- VALUE 

SIG 

N* 

DEVIATION* 

13 

50.0% 

53.6% 

1.44 

NS 

22 

22 

9.7 

6.6 

23 

56.0% 

54.0% 

.31 

NS 

5 

4 

13.0 

9.0 

LL 

49.9% 

57.7% 

2.17 

P< .  05 

14 

17 

12.3 

7 . 7 

6Y 

60.1% 

54.7% 

1.04 

NS 

17 

10 

7.0 

15.6 

3M 

50.3% 

52.5% 

.97 

NS 

24 

25 

8.9 

6.7 

43 

44.5% 

53.8% 

1.49 

NS 

17 

16 

10.1 

16.7 

5B 

55.3% 

54.8% 

.08 

NS 

4 

4 

8 . 5 

10.1 

*  The  first  number 

represents 

the  control 

group  while 

the  second 

number  represents  the  experimental  group. 


TRADOC  expanded  the  TRAC  analysis  to  overall  comparison  of 
experimental  and  control  groups,  the  results  of  which  follows  in 
TABLE  3.  The  two  groups  are  the  same  with  the  exception  of  the 
intervention  of  JSEP  lessons  on  the  experimental  group.  We  found 
a  significant  difference  (C.05)  between  the  experimental  and 
control  groups.  Scores  reflected  in  the  Mean  column  represent  the 
percent  of  answers  correct  on  the  ARI  Job-Relevent  Knowledge 
Tests.  We  equalized  the  seven  MOSs  in  the  experimental  group  by 
using  percent  of  answers  correct  since  the  number  of  questions  on 
each  of  the  seven  ARI  MOS  Tests  varied.  The  experimental  group  in 
TABLE  3  demonstrates  a  decline  in  group  variance  as  represented  by 
the  standard  deviation  (S.D.  10.22). 


TABLE  3 


FIRST  ARI  JOB-RFLEVANT  KNOWLEDGE  TEST 


GROUP 

N 

MEAN 

S.D. 

MEAN  DIF. 

t  = 

Exper imental 

98 

54.21 

10.22 

3.27 

2.06* 

Control 

103 

50.94 

12.17 

*  Significant  at  <.05  level  (?=0.041). 


Table  4  demonstrates  the  results  of  the  second  ARI  Test  120 
days  after  AIT/OSUT.  Data  generated  from  the  second  test  provided 
the  information  we  needed  for  the  retention  portion  of  our 
hypothesis.  ESOs  throughout  the  Army  were  able  to  contact  about 
half  of  each  group  for  the  second  test.  We  found  13  soldiers  from 
our  groups  had  separated  from  the  Army  by  the  time  we  queried  the 
Enlisted  Master  File  at  DA  for  permanent  duty  station  addresses. 


TABLE  4 

SECOND  ARI  JOB-RELEVANT  KNOWLEDGE  TEST 


N 

MEAN 

S.D. 

Experimental 

50 

57.16 

8.41 

Comparison  * 

47 

59.68 

9.03 

*  The  use  of  the  term  "comparison"  recognizes  loss  of  control 
group  characteristics  through  experimental  mortality. 

Beyond  the  first  posttest,  there  was  no  attempt  to  equalize 
differences  between  groups. 


As  demonstrated  in  TABLE  5,  we  divided  the  Experimental  Group 
into  three  subgroups  in  order  to  analyze  the  effects  of  the  number  of 
JSEP  lessons  on  soldiers  since  participants  did  not  have  the  benefit 
of  completing  all  of  their  individual  prescription  assigned  by  the 
JSEP  Management  System.  The  percentage  of  lessens  completed  is  by 
each  Subgroup  is  as  follows:  Subgroup  1  =  96  to  100%;  Subgroup  2 
-  86  to  95%;  Subgroup  3  -  85  to  21  %.  The  shaded  area  represents 
the  theoretical  regression  of  the  comparison  group. 

Also,  TABLE  5  displays  the  comparison  of  the  three  subgroups 
that  make  up  the  experimental  group  and  shows  the  improvement 
among  the  groups  from  Test  1  to  Test  2.  Positive  slope  of  all 
experimental  subgroups  demonstrates  increase  of  JSEP-like  skills 
in  the  work  environment.  Such  an  increase  indicates  a  high 
content  validity  of  the  JSEP  curriculum. 

TABLE  5 


COMPARISON  OF  SUBGROUPS  TEST  1  (TI)  TO  TEST  2  (T2) 
Membership  Determined  by  %  of  Prescription  Completed 


We  performed  a  One-Way  ANOVA  on  the  experimental  groups, 

(TABLE  6) ,  that  represents  the  two  test  administrations, 

Tl  and  T2 ,  approximately  120  days  apart.  The  significance  of  the 
ANOVA  demonstrates  that  group  completion  of  all  or  nearly  all  of  the 
prescription  contributes  greatly  to  homogeneity,  and  conversely, 
completing  fewer  prescribed  lessons  (or  none)  increases  ingroup 
variance.  Another  significant  finding  is  that  differences  between  the 
subgroup  completing  all  or  nearly  all  of  the  prescription  and  other 
subgroups  grew  over  time  beyond  the  intervention.  As  demonstrated  by 
the  highly  significant  finding  at  T2 /  the  effects  of  JSEP  last  well 
beyond  the  initial  impact  of  the  instruction. 


TABLE  6 
ONE-WAY  ANOVA 

_ MEAN _  MS  ERROR  F  (2 . 47) 

Group  1  Group  2  Gr oup3 

Test  1  58.41  53.47  55.44  223.49  85.60  2.61  * 

Test  2  61.76  54.18  55.44  279.70  61.86  4.52  ** 

*  Significant  at  P=0.08  (marginal  finding) 

**  Significant  at  P=0.02 


CONCLUSION 


Our  hypotheses  that  JSEP  is  a  cost  effective  method  of 
soldiers  acquiring  and  retaining  job  related  basic  skills  were 
proven  true.  Our  findings  indicate  that  soldiers  who  participate 
in  JSEP  will  out  perform  nonparticipants  on  targeted  skills. 
Secondly,  the  demonstrated  effects  of  job  skills  training  will  be 
higher  in  targeted  skills  with  JSEP  preparation.  Third,  the 
effect  of  JSEP  ties  directly  to  the  percent  of  lessons  completed 
by  the  soldier. 

Soldiers  who  have  the  benefit  of  JSEP  instruction  and  are 
allotted  the  time  to  complete  the  entire  prescription  of  JSEP 
lessons  assigned,  do  perform  significantly  better  in  job  tasks  and 
continue  to  make  significant  gains  as  they  apply  that  knowledge  to 
the  job  setting  over  time  than  soldiers  who  do  not  take  JSEP. 

There  is  a  significant  positive  relationship  between  the  number  of 
JSEP  lessons  completed  and  job  performance  of  soldiers  as  noted 
between  groups  in  our  Test  Plan  (experimental  and  comparison  as 
well  as  within  the  three  subgroups  of  experimental  soldiers). 

Considering  the  results  of  the  JSEP  Test  Plan,  and  the 
results  of  the  JSEP  Test  for  Artillery  Career  Management  Field 
(CMF  13)  concluded  in  July  1990,  we  support  and  will  recommend 
that  learning  centers  with  PLATO  capabilities  adopt  the  JSEP 
curriculum  as  the  primary  mode  for  delivering  basic  skills  lessons 
and  the  Army  pursue  alternate  CBI  delivery  systems  for  JSEP. 
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